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.20.  Abstract  cont.  i. 

It  is  hoped  that  this  idea  will  help  to  clarify  the  relationship  between 
splines  and  generalized  inverses. 

In  (2)  the  investigator  pointed  out  the  relationship  between  Lard’s  series 
'representation  of  the  generalized  inverse  and  Kryanev's  iterative  method  for 
solution  of  anoperator  equation  of  the  first  kind  with  a  closed  unbounded 
operator.  Under  certain  assumptions  on  the  data  an  error  bound  is  alsos  estab 
lished . 

Convergence  theorems  and  error  bounds  for  a  very  general  class  of  regulari¬ 
zation  methods  are  developed  in  (3).  The  error  bounds  related  the  smoothness 
of  the  data  to  a  modulus  of  convergence  for  the  general  regularization  method 
(which  includes  as  special  cases  both  iterative  and  noniterative  regularizatio 
methods).  Work  is  in  progress  on  inverse  results  which  show  that  a  certain 
rate  of  convergence  for  a  regularization  method  implies  a  certain  degree  of 
smoothness  for  the  data. 

Paper  (A)  surveys,  extendds  and  unifies  the  mathematical  theory  of  parameter 
choice  in  linear  regularization.  Included  are  discussions  of  a  priori  para¬ 
meter  choice  strategies,  the  Discrepancy  Principle,  the  derivative  and  ratio- 
criteria,  and  a  parameter  choice  criterion  for  approximating  constrained  pseud 
solutions. 
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A  number  of  results  on  general  regularization  methods  for 
ill-posed  linear  problems  and  related  mathematical  ideas  were 
developed  during  the  course  of  the  project.  These  results  are 
documented  in  four  papers,  all  of  which  have  been  accepted  for 
publication.  The  four  papers  and  the  journals  in  which  they  will 
be  published  are  as  follows: 

1.  Generalized  inverses  and  generalized  splines;  Numerical 
Functional  Analysis  and  Optimization. 

2.  On  the  Kryanev-Lardy  method  for  ill-posed  problems; 
Mathematische  Nachrichten. 

3.  On  a  class  of  regularization  methods;  Bolletino  della 
Uni one  Matematica  I tali  ana. 

4.  The  parameter  choice  problem  In  linear  regularization: 
a  mathematical  introduction,  in  "Ill -Posed  Problems: 
Theory  and  Practice"  (M.Z.  Nashed,  Ed.),  Wiley,  New  York. 


The  investigator  has  taken  the  point  of  view  of  generalized 
inversion  in  studying  numerical  methods  for  the  regularization 
of  linear  ill-posed  problems.  In  paper  (1)  some  ideas  of 
A.  Sard  are  extended  to  provide  a  general  axiomatic  framework 
for  both  splines  (including  interpolator  splines  and  generalized 
harmonic  functions)  and  the  Moore-Penrose  generalized  inverse.  It 
is  hoped  that  this  idea  will  help  to  clarify  the  relationship 
between  splines  and  generalized  inverses. 

In  (2)  the  investigator  pointed  out  the  relationship  between 
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Lardy's  series  representation  of  the  generalized  inverse  and 
Kryanev's  iterative  method  for  solution  of  an  operator  equation 
of  the  first  kind  with  a  closed  unbounded  operator.  Under  certain 
assumptions  on  the  data  an  error  bound  is  also  established. 

Convergence  theorems  and  error  bounds  for  a  very  general 
class  of  regularization  methods  are  developed  in  (3).  The  error 
bounds  relate  the  smoothness  of  the  data  to  a  modulus  of  convergence 
for  the  general  regularization  method  (which  includes  as  special 
cases  both  iterative  and  noniterative  regularization  methods). 

Work  is  in  progress  on  inverse  results  which  show  that  a  certain 
rate  of  convergence  for  a  regularization  method  implies  a  certain 
degree  of  smoothness  for  the  data. 

Paper  (4)  surveys,  extends  and  unifies  the  mathematical 
theory  of  parameter  choice  in  linear  regularization.  Included 
are  discussions  of  a  priori  parameter  choice  strategies,  the 
Discrepancy  Principle,  the  derivative  and  ratio  criteria,  and  a 
parameter  choice  criterion  for  approximating  constrained 
pseudo-solutions. 

Complete  copies  of  the  papers  follow  as  appendixes. 
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ABSTRACT 


An  abstract  framework  in  Hilbert  space  is  provided  for  gen¬ 
eralized  splines  and  generalized  inverses  of  operators. 


Since  Atteia  [1]  introduced  an  abstract  point  of  view 
in  the  theory  of  spline  functions,  a  number  of  authors  have 
studied  abstract  splines  in  Hilbert  space  and  relationships 
between  spline  functions  and  generalized  inverses  (see  [2], 
[3],  [4],  [6],  [7]).  In  [7]  Sard  developed  a  very  elegant 
theory  of  "splines"  in  Hilbert  space.  Our  aim  in  this  brief 
note  is  to  enlarge  somewhat  the  context  introduced  by  Sard 
so  as  to  provide  a  framework  which  encompasses  the  splines 
in  the  sense  of  Sard  and  also  the  concept  of  the  generalized 
inverse  of  a  linear  operator  in  Hilbert  space. 

Suppose  that  X  and  W  are  linear  spaces  and  that  Y  and  Z 
are  inner  product  spaces  (the  inner  product  and  induced  norm 
in  each  space  will  be  denoted  by  <•,•>  and  ||*||,  respec¬ 
tively).  We  also  assume  that  there  are  two  linear  mappings 

F  .•  X  x  W  -  Z  and  U  :  X  x  W  -*•  Y 


which  satisfy 


(1)  F(x.O)  =  0  and  U(x,0)  =  0  implies  x  =  0 


and 

(2)  F(0,w)  =  0  and  U(0,w)  =  0  implies  w  =  0. 

These  linear  maps  are  the  bivariate  analogues  of  Sard's  "observa¬ 
tion"  and  "coobservation"  operators  {see  [7]).  By  virtue  of 
(1),  the  bilinear  form 

[x,y]  «  <F(x,0),F(y,0)>  +  <U(x,0) ,li(y,0)> 

is  an  inner  product  on  X  and  we  will  denote  by  7  the  Hilbert 
space  which  is  the  completion  of  X  in  the  norm  |*|  induced 
by  this  inner  product.  Condition  (2)  guarantees  that  the  bi¬ 
linear  form 

(w,v)  =  <F(0,w),F{0,v)>  ♦  <U( 0,w),U(0,v)> 

is  an  inner  product  on  W  and  we  shall  designate  the  completion 
of  W  with  respect  to  this  inner  product  by  W.  Note  that  the 
mappings  F  and  U  are  continuous  on  X  x  W  and  we  will  persist 
in  denoting  the  continuous  extensions  of  these  mappings  to 
7  x  W  by  F  and  U  respectively.  We  assume  that  for  each  b  c  W 
the  closed  convex  set  p(b)  defined  by 

p(b)  =  {x  e  7  :  F(x,b)  *  0} 

is  nonempty  (note  that  p(0)  is  a  closed  subspace  of  7).  One 
may  view  this  requirement  as  an  abstract  "interpolation" 
condition.  The  symbol  will  designate  the  subspace  of  W 
which  is  maximal  (relative  to  inclusion)  with  respect  to  the  pro¬ 
perty  that  p(b)  f  0  for  all  b  c  W.  Note  that  W  c  W  c  w.  As  a 
final  bit  of  notation,  w  will  be  the  mapping  which  associates 
with  each  closed  convex  subset  of  7  Its  unique  element  of 
minimal  norm. 


Defini tion.  The  mapping  b  ■*  b+  from  W  into  7  defined  by  b+ 
nop(b)  will  be  called  the  generalized  spline  mapping  as¬ 
sociated  with  the  structure  (X,W ,F,U). 

Note  that  if  we  set  M  =  pfO)1',  then  one  sees  readily 
that  p(b)  =  b+  +  p(0)  and  M  n  p(b)  =  {bf}  . 

Proposition  1.  If  b  e  W,  then  ||U(x,0)||  is  minimal  among 
all  x  e  p(b)  if  and  only  if  x  =  b+. 

Proof.  If  x  e  p(b),  then  since  F(x,b)  =  0  and  F(b+,b)  =  0, 
we  find  that  x  -  b+  e  p(0)  =  M"*".  Therefore 

IIU(X,0)I12  +  | |F(x,0) I  I2  =  |x|2  =  | X  -  b+|2  +  |b+|2. 

But  since  F(x  -  b+,0)  =  0,  it  follows  that  F(x,0)  *  F(b+,0) 
and  hence 

||U(xt0)||2  »  | X  -  b+|2  ♦  |b+|2  -  ||F(b+,0)||2, 
which  establishes  the  assertion. 

Proposition  2.  The  generalized  spline  mapping  b  -*  b+  is  a 
closed  linear  operator  which  is  continuous  if  and  only  if  W 
is  complete. 

Proof.  The  linearity  follows  easily  from  the  fact  that 
p(bj  +  b2)  *  {x  +  y  :  x  e  p(b^),  y  c  p(b2)}  and  the  rep¬ 
resentation  p(b)  «  b+  +  p(0).  Suppose  that  {bn >  c  ft  and 
(bn,bn)  (*»b)  c  X  x  W.  By  the  continuity  of  F,  we  then 


F(x,b)  -  11m  F(b;,bn)  -  0. 
n 


Therefore  x  e  p (b )  and  hence  b  e  W,  by  the  maximal ity  of  W. 

Also,  since  M  is  closed  and  {b+}  C  M,  we  have  x  c  M  C\  p(b)  = 
t  ^ 

(b  }.  Therefore  the  graph  of  the  generalized  spline  mapping 
Is  closed,  that  is,  the  generalized  spline  mapping  is  a 
closed  operator. 

If  W  is  complete  then  the  generalized  spline  mapping  is 
continuous  by  the  Closed  Graph  Theorem.  On  the  other  hand, 
if  the  generalized  spline  mapping  is  continuous,  then  it  has 
a  continuous  extension  b  -*■  defined  for  all  b  e  W.  Suppose 
b  e  W  and  choose  a  sequence  {bn >  C  W  with  bn  -*•  b.  Then, 
since  F  is  continuous, 

F(b#,b)  =  lim  F(bJ.bn)  =  lim  F(bJ,bn)  =  0. 
n  n 

Therefore  p(b)  ?  0  and  hence  b  c  Q.  It  follows  that  W  c  w, 
that  is,  W  is  complete. 

In  the  special  case  when  W  =  X,  F(x,b)  *  Fx  -  Fb  and 
U(x,b)  =  Ux  -  Ob,  where  F  and  U  are  linear  operators,  we 
recover  Sard's  theory  of  splines.  Here  W  =  X,  M  =  N(F) 
is  the  space  of  "splines"  and  b+  =  P^b,  the  projection  of  b 
onto  M,  is  the  spline  approximation  to  b  c  X.  Proposition 
1  is  then  just  a  statement  of  the  "optimal  interpolation" 
property  of  splines.  For  specific  applications  to  inter¬ 
polator  splines  and  generalized  harmonic  functions  see 
[7]. 

As  another  example,  suppose  and  H2  are  Hilbert  spaces, 
D(T)  is  a  dense  subspace  of  and  T  :  D(T)  -+  H2  is  a  closed 
linear  operator.  Let  Q  be  the  orthogonal  projection  of 
H2  into  FRTT.  Let  X  =  D(T),  W  =  R(T}  +  R(T)X  ,  Z  =  «2  and 
Y  *  H.j  x  H2.  Define  the  linear  operators  F  :  X  x  W  -*■  Z 
and  U  :  X  x  W  -*•  Y  by  F(x,b)  =  Tx  -  Qb  and  U(x,b)  =  (x,b). 

In  this  case  X  *  D(T)  ,  W  =  W,  p{b)  is  the  set  of  least 
squares  solutions  of  the  equation  Tx  *  b  and  b+  *  T+b, 
where  Tf  Is  the  Moore-Penrose  inverse  of  T  (see  e.g.  [5]). 


Proposition  1  in  this  case  expresses  the  well-known  extremal 
property  of  the  Moore-Penrose  inverse  and  Proposition  2  is 
the  (somewhat  less)  well-known  characterization  of  continuous 
Moore-Penrose  inverses. 
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On  the  Kryanev-Lardy  Method  for  Ill -Posed  Problems* 

C.  U.  Groetsch 

1.  Introduction. 

Kryanev's  method  for  solving  the  Ill-posed  operator  equation 

(1)  Au  -  f 

where  A  Is  a  linear  operator  on  a  real  Hilbert  space,  consists  of 
choosing  a  bounded,  positive  definite  operator  B  and  forming  the 
sequence  of  iterates  defined  by 

(2)  x0  "  0  »  +  Bxn  “  Bxn-1  +  f 

Krayanev  [3]  established  the  convergence  of  the  method  under  the 
assumption  that  A  Is  a  bounded  positive  semi-definite  operator  and 
equation  (1)  has  a  unique  solution.  The  author  [2]  proved  the  convergence 
of  a  related  method  In  the  case  when  A  Is  a  densely  defined  closed  linear 
operator,  again  under  the  assumption  that  for  a  given  f  equation  (1)  has 
a  unique  solution.  Our  aim  in  this  note  is  to  investigate  the  convergence 
of  the  method  to  a  generalized  solution  of  (1)  when  A  is  a  closed 
unbounded  operator  and  the  existence  of  a  unique  solution  is  not  assumed. 

f Partially  supported  by  AFOSR  Grant  79-0059 
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2.  Results. 

Suppose  that  and  H2  are  real  Hilbert  spaces  (the  Inner  product 
In  each  space  will  be  designated  by  <•,♦>)  and  that  D(A)  is  a  dense 
subspace  of  .  Let  A  :  D( A)  -►  H2  be  a  closed  linear  operator.  We  shall 
investigate  an  Iterative  method  for  approximating  A+f,  where  Af  is  the 
Moore-Penrose  generalized  inverse  of  A.  We  recall  that  A+  is  the 
closed  linear  operator  defined  on  the  dense  subspace 

D(A+)  ■  R(A)  9  R(A)X 

of  H2  by  A+f  ■  u,  where  u  is  the  solution  of  minimal  norm  of  the  equation 

(3)  Ax  =  Qf  , 

and  Q  is  the  orthogonal  projection  of  H2  onto  R(A)'.  We  note  that  this 
definition  of  the  generalized  inverse  is  the  same  as  that  for  a  bounded 
operator  (see  e.g.  [1])  and  that  for  f  c  D(A+)  the  set  of  solutions  of  (3) 
is  convex,  nonempty  and  closed  (since  A  is  closed).  Therefore  the 
vector  A+f  is  uniquely  defined. 

We  shall  suppose  that  B  :  H.j  -*■  Hj  Is  a  bounded,  self-adjoint 
operator  satisfying 

(4)  <Bx,x>  >  cZ  |  |x 1 12 

\ 

for  all  X(H|  and  some  c  f  0.  We  may  define  an  equivalent  Inner  product 


f 


\ 

\ 


on  by 

[x,y]  =  <Bx,y>  , 

and  we  shall  denote  the  Hilbert  space  which  consists  of  H1  with  the  new 
inner  product  [•»•]  by  The  norm  in  H ^  will  be  denoted  ||*||g,  that 
is 


2 

| | x | |g  -  <Bx,x>  =  [x,x]  . 

Since  the  norms  (HI  and  I !  *  1 1 B  are  equivalent,  the  subspace  D(A)  is 
also  dense  in  Hy  The  adjoint  of  A  considered  as  an  operator  on  H ^  will 
be  designated  by  A'.  That  is, 

A’  :  D(A')  -  tf1 

satisfies  <Ax,y>  ■  [x.A'y] 

for  all  y  e  D(A* )  *  {y  e  Hg  :  <Ax,y>  *  [x,z],  some  z  e  and  all  x  c  D(A)>. 
The  adjoint  of  A  considered  as  an  operator  on  will  be  designated  by  the 
customary  symbol.  A*.  Note  that  these  two  adjoints  are  related  in  a 
simple  way.  Namely,  0(A‘)  ■  D(A*)  and  A*  ■  BA*.  By  [5,  page  307]  the 
operator 

+  A*  A)-1  , 
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where  Ij  is  the  identity  operator  on  is  a  bounded,  self-adjoint 
operator  on  H ^  satisfying 

(5)  +  a'a)”*1  lB  i  1  • 

Lemma  1.  Given  f  e  Hg,  there  is  a  unique  Uj  e  0(A)  such  that 

<BUi,v>  +  <Au^,Av>  *  <f,Av> 

for  all  v  c  0(A) .  Moreover,  if  f  e  D ( A+ ) ,  then  u-j  ■  u  -  Wu,  where 
W  =*  +  A’A)"1  and  u  -  Aff. 

Proof.  Since  A  is  closed,  0(A)  is  a  Hilbert  space  under  the  inner  product 
(•,•)  defined  by 


(x,y)  -  [x,y]  +  <Ax,Ay>  . 

The  linear  functional  ♦  defined  on  D(A)  by  $(v)  »  <f,Av>  |s  clearly 
continuous  with  respect  to  the  norm  Induced  by  the  inner  product  (•,•)• 
Therefore,  by  the  Riesz  Theorem,  there  Is  a  unique  e  D(A)  with 

<f,Av>  -  *(v)  ■  (uj.v) 

for  all  v  e  0(A),  which  was  to  be  shown. 

If  f  e  D(A+)  and  u  ■  A+f,  then  Au  ■  Qf,  and  since  <f,Av>  ■  <Qf,Av> 
for  all  v  c  0(A)  and  Wu  c  D(A*A),  we  have 
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<B (u  -  Wu),v>  +  <A(u  -  Wu),Av> 

■  <Bu,v>  -  <BWu,v>  -  <AWu,Av>  +  <Au,Av> 

■  <Bu,v>  -  <(B  +  A*A)Wu,v>  +  <Qf,Av> 

■  <Bu,v>  -  <B(I-j  +  A'A)Wu,v>  +  <f,Av>  *  <f,Av>  . 

But  then,  by  the  first  part  of  the  Lemma,  u-j  *  u  -  Wu,  which  was  to  be 
proved. 


We  will  study  the  sequence  of  Iterates  defined  by 

(6)  un  •  Wun  .j  +  Uj  ,  n  ■  1,2,3,... 

where  u^  Is  given  by  Lemma  1.  Note  that  this  Is  equivalent  to  the 
requirement  that 


<Bu  ,v>  +  <Au„,Av>  ■  <Bu„  , ,v>  +  <f,Av> 
n  n  n-i 

for  all  v  e  0(A) ,  which  establishes  the  connection  with  Kryanev’s  method. 
This  can  be  established  exactly  as  In  the  proof  of  Lemma  1  by  considering 
the  linear  functional 


♦(v)  ■  <Bun_1,v>  +  <f,Av> 
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on  the  Hilbert  space  D(A).  In  the  case  B  ■  I  and  f  e  D(A+), 
(6)  reduces  to  that  investigated  by  Lardy  [4]. 

Lemma  2.  If  f  e  D(A+)  and  u  ■  A+f,  then  u  -  un  *  Wnu. 

Proof.  By  (6)  and  Lemma  1,  we  have 

u  -  un  ■  u  -  u-j  -  Wun_.j  ■  u  -  (u  -  Wu)  -  Wun_.j 

■  W(u  -  un-1)  . 


Therefore,  u  -  un  ■  Wnu,  n  ■  1,2,3,...  . 

We  may  now  provide  an  error  bound  for  the  method  (6). 
we  will  henceforth  denote  the  operator  A*A  by  A. 

Theorem  1.  Suppose  R(A)  c;  R(A'A)  and  Qf  ■  AAz  for  some  z  e 
some  y  e  Hj, 

lb  -  u„IIb  i?n“T  ‘  n*n  My llj  •  for  n  >  i  . 

Proof.  Since  R(A)  ^R(A'A),  we  have  Az  ■  A'Ay,  for  some  y  i 
(Ex }  be  the  resolution  of  the  identity  in  H ^  induced  by  the 
operator  A'A.  Since  Qf  ■  AAz  and  Az  c  N(A)J\  we  have  Az  ■ 


u  -  u  -  V^A'fy  ■  / - r-  dE.y 

n  0(1  ♦  X)n  X 


the  method 


For  convenience 

H.| ,  then  for 


H1 .  Let 
self-adjoint 
+f.  Therefore 


by  Lenina  2, 
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It  then  follows  that 


11--01 


0  (1  +  x)‘ 


dCE^.y] 


in4r  (’  -  b"  Ac^y.y] 

■h4t(1  -n>"  i  iyi  Ib  • 


We  note  that  if  we  make  the  weaker  assumption  that  f  e  D(A*)  (i.e. 

Qf  e  R(A)),  rather  than  the  stronger  assumption  that  Qf  e  R(AA) ,  then  the 
method  still  converges.  For  In  this  case  we  have  by  Lemma  2, 

llu  -  unllB  -  /  ( 1  +  X)’nd[E?u,u]  -*-0  as  n  . 

However,  If  f  i  D(A+),  then  the  sequence  (un >  diverges  and  in  fact  has  no 

weakly  convergent  subsequence.  For  if  the  subsequence  (u  }  converges 

nk 

weakly  to  y,  then  since  W  Is  bounded  and  therefore  weakly  continuous,  we 
have  by  (6) 

(7)  y  -  Wy  ■  u1 

Since  u<|  c  0(A)  and  Wy  c  D(A)c.D(A),  we  find  that  y  c  0(A).  Also,  by 
Lemma  1  and  (7) 
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i; 

| '  <f ,Av>  ■  <8y  -  BWy,v>  +  <Ay  -  AWy,Av> 

-  <By,v>  -  <{B  +  A)Wy,v>  +  <Ay,Av> 

*  <Ay,Av>  ,  for  all  v  c  D(A) 

|  Therefore,  f  -  Ay  e  R(A)  ,  that  Is,  f  e  D(AT).  We  summarize  these  results 

|  in  the  following: 

1  Theorem  2.  If  f  e  D(A+),  then  un  A*f.  However,  if  f  t  D(A+),  then 

j  (un)  has  no  weakly  convergent  subsequence. 

Since  bounded  sets  in  Hilbert  space  are  weakly  compact,  we  obtain 
immediately  the  following: 

» 

j  Corollary.  If  f  i  D(A+),  then  | | un | |  -*■  <». 

1  Finally,  we  Investigate  the  method  under  the  assumption  that  the  exact 

j  data  f  Is  unavailable,  but  an  approximation  f  satisfying  | | f  -  f ||  <_  6  is 

I  on  hand.  The  first  approximation  u^  (corresponding  to  the  corrupted  data 

j  ?)  then  satisfies 

(uj,v)  ■  <Buj,v>  +  <Aiij,Av>  »  <f,Av> 

i 

for  all  v  c  D(A).  By  Lemma  1  we  then  have 
j  (u^  -  u] ,v)  ■  <f  -  f ,Av> 
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for  all  v  e  D(A).  Setting  v  -  -  u^,  this  gives 

(8)  (u1  -  iij,  Uj  -  Uj)  *  <f  -  f,  A(u1  -  u1)>  . 

If  we  designate  the  norm  on  0(A)  Induced  by  the  inner  product  (•»•)  by 
1 1*  1 1 1 •  then  we  have 

2  2 
I |A*| \  £  <Bx,x>  +  <Ax,Ax>  ■  | |x| 

for  all  x  c  0(A) .  Therefore  by  (8),  it  follows  that 

ll“,  H*tul  i*llui  -'“ill) 

and  hence 

I  lu!  *  ^Jlg  <  |  |u1  -  GlMl  <6. 

Later  approximations  using  the  data  f  satisfy  by  (6) 

"n  ’  “Vl  *  V 


“»*“»■  X"k(ul  •  “l> 


1,2,3 . 


and  therefore 
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But  ||W||B  <_1,  by  (5),  and  therefore 

Mun  -  “nllB  i  nMul  “  “ills  ^n6* 

It  then  follows  that 

IK  -  UHB  ^  IK  -  “nils  +  Hun  “  UHB  -nfi  +  Hun  "  uNb* 

But  |  |un  -  u [  | B  0  as  n  +•  If  f  e  D(A+).  Therefore,  given  e  >  0  there  is 

a  6(c)  and  n  such  that 
e 

1 |un  -  u| |B  <  e  for  0  <  6  <  6(e), 

that  is,  the  method  is  a  regularizing  algorithm  in  the  sense  of  Tikhonov 
(see  [6])  If  f  e  D(A+). 
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Zusammenfassung 

Wir  studieren  eine  allgemeine  Klasse  von  Regularisierungsmethoden 
fur  eine  inkorrekt  gestellte  lineare  Operatorglei chung  im  Hilbertsraum. 
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On  a  Class  of  Regularization  Methods* 

C.  W.  Groetsch 

In  this  note  we  will  Investigate  a  general  class  of  regularization  methods 
for  the  ill -posed  operator  equation 

(1)  Tx  =  b 

where  T  is  ;  hounded  linear  operator  from  the  Hilbert  space  H-j  into  the  Hilbert 
space  H^.  The  Moore-Penrose  generalized  inverse  of  T  will  be  denoted  by  T+,  that 
is  T  :  P(T  )  -►  H-j  is  the  linear  operator  which  associates  with  each  vector 
b  e  P(T+)  :  -  R(T)  ®  R( T)  the  unique  least  squares  solution  of  minimal  norm  of 
equation  (1)  (see  e.g.  [3]).  By  a  least  squares  solution  of  (1)  we  mean  any 
solution  of  the  equation 

.  *  * 

(2)  T  Tx  =  T  b, 

*  * 
where  T  is  the  adjoint  of  T.  We  will  denote  the  operator  T  T  by  T  and  -he  operator 

*  - 

TT  by  T.  Note  that  T  and  T  are  self-adjoint  linear  operators  whose  spectra  lie  in 
the  Interval  [0, ||T||2].  If  0  i  o(T)  (the  spectrum  of  T),  then  by  (2)  we  have 
T+  *  T_1T*.  In  general,  however,  0  e  a(T),  but  this  last  equation  nevertheless 
leads  us  to  seek  approximations  to  T+  by  operators  of  the  form  U(T)T*  where  U  is  a 
continuous  function  on  [0,||T||  ]  which  approximates  the  function  f(t)  =  t  in 
some  sense.  Specifically,  we  will  consider  a  family  (net)  of  real  valued  functions 
(U0(t)  :  8  c  S),  Indexed  by  a  subset  S  of  the  positive  real  numbers  with  «  e  S, 

P 

2 

where  each  Ug  Is  continuous  on  [0,||T||  ]  and  such  that 
^Partially  supported  by  AFOSR  grant  79-0059. 


2 


(3) 

|tU0(t)|  <  M 

for  all  t  and  & 

(4) 

Ug(t)  -  t'1 

as  B  ->  »  for  each  t  i1  0 

Such  a  class  of  regularization  methods  for  equation  (1)  was  studied  previously 
by  Bakushinskii  [1]  under  the  assumption  that  b  e  R(T).  (The  author  was  unaware 
of  Bakushinskii 's  studies  when  this  research  was  performed.) 

The  following  is  proved  in  [3]. 

Proposition  1.  Suppose  b  e  D(T  )  and  let  xg  *  Ug(T)T  b.  Then  xg  -  T  b  as  e  -*>  ». 
To  this  we  now  add. 

Proposition  2.  If  b  t  P(T+),  then  {xg}  has  no  weakly  convergent  subnet  and 
hence  1 1  |  j  «  as  e  -*■  ®. 

Proof.  Suppose  {xg , >  is  a  subnet  of  {xg}  which  converges  weakly  to  z  e  H1 ,  de- 
w 

noted  x0,  -*■  z.  By  the  weak  continuity  of  bounded  linear  operators  we  then  have 

P 

Txgl  *  Tz.  Now,  if  we  denote  the  projection  of  H2  onto  R(t7  by  P,  then 

Pb  -  TXg  =  Pb  -  TU8(T)T*b 
=  Pb  -  TU_ ( f ) Pb . 

However,  by  (3)  and  (4),  the  operator  TIL(T)  converges  pointwise  to  the  projection 
-  x  *  x  . 

of  H2  onto  N(T)  *  N(T  )  ■  Rff!-  Therefore  Pb  -  Txg,  -*•  0.  It  then  follows  that 

Pb  *  Tz,  a  contradiction.  # 

In  the  proof  above  we  have  used  the  fact  that  Ug(T)T*  *  T*Ug(T).  This  is 
easy  to  see  if  Ufl  is  a  polynomial.  In  the  general  case  the  identity  follows  from 

M 

the  Welerstrass  approximation  theorem. 
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Several  authors  have  established  rates  of  convergence  for  various  approx¬ 
imations  to  T+b  under  the  stronger  assumption  that  Pb  e  R(T)  (see  [9],  [5], 
[6]).  We  see  from  Proposition  2  that  the  very  least  we  must  require  to  get 
convergence  at  all  is  that  b  e  P(T+),  i.e.,  Pb  e  R(T).  In  order  to  strengthen 
this  condition  only  slightly  and  thereby  obtain  a  rate  of  convergence  we  note 
that 

R(T)  =  R(TPn(t)a) 

and,  in  the  pointwise  sense. 


PN(T) 


x 


1 im  Tv. 
v->0 


It  therefore  seems  reasonable  to  replace  the  hypothesis  b  e  p(T+),  i.e.,  Pb  e  R(T) , 
by  the  hypothesis  Pb  e  R(TTV)  for  some  v  >  0.  In  order  to  gauge  the  rate  of  con¬ 
vergence  we  will  replace  (3)  by  the  stronger  condition 


(5)  tv 1 1  -  tUQ(t)|  <  u>(e,v)  for  V  >  0 

P 

where  o>(e,v)  0  as  M  «  for  each  v  >  0  (the  case  v  =  1  was  considered  in  [4]). 

The  proof  of  the  following  lemma,  being  routine,  is  omitted. 

Lemma  1.  If  v  >  0,  then  R(Tv)c  N(T)J\ 

We  now  state  a  rate  of  convergence  result.  The  vector  T+b  will  be  denoted 
by  x  and  the  error  x  -  x.  by  e0. 

P  P 

Proposition  3.  If  Pb  *  TTvw,  where  v  >  0,  then  I  |eBl  I  £  <o(6,v)  |  |w 1 1 . 

Proof.  Since  Tx  ■  Pb  ■  TTvw  and  since  x  -  Tvw  e  N(T)**’ •  we  see  that  x  ■  Tvw.  Now, 

XB  *  Ug(T)T*b  -  Ug(T)T*Pb 
»  U0(T)fx  »  U0(T)Tv+1w. 


Therefore  e  =  x  -  x.  s  Tv ( I  -  U0(T)T)w.  By  the  Spectral  Mapping  Theorem 

P  P  P 

and  Radius  Formula,  we  then  have 

I  |eg|  |  i  u{s,v)| |w| |.  # 

In  our  next  result  we  become  more  cavalier  in  our  assumptions  on  the  data. 

Lemma  2.  If  Pb  =  Tvw  where  v  >_  1 ,  then  | |eg| |2  <  u>(e,v-l)| | Te^ [ [  | |w| |. 

Proof.  As  in  the  previous  proof  we  find  that  x  ■  T*Tv"^w.  Also, 

xe  =  Ue(T)T*Pb  =  Ue(T)T*Tvw 
-  T*ue(T)Tvw. 

Therefore  e0  *  x  -  xg  =  T*(I  -  Ug(T )T)Tv_1w,  and 

||e6||2  =  (e0,T*(I  -  Ug(T)T)Tv-1w 

=  (Teg,(I  -  UB(T)T)Tv-1w)  <  a>(8,v-l)  |  |w|  |  ||Te6||.  # 

«  o 

Proposition  4.  If  Pb  =  Tvw  where  v  ^  1 ,  then  1 1 e0 1 1  <_  w(b,v)o)(3,v-1  )  |  |w|  | . 

Proof.  In  Lemma  2  we  saw  that 

eB  *  T*(I  -  U0(T)T)Tv_1w, 

therefore 

Te0  -  tV(I  -  U0(T)T)w. 

We  then  have 

l|Teell2  -  (fe6,ee)  ■  <TV(I  -  U#<T)T..Tee) 

<  u(6,v) | |Teg| |,  i.e.,  | |Teg j |  <.  u(3,v) . 

Substituting  Into  the  result  of  Lemma  2  completes  the  proof.  # 
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In  the  next  section  we  will  give  a  number  of  examples  of  specific  com¬ 
putational  techniques  to  which  the  above  results  apply. 

We  have  avoided  for  long  enough  the  problem  of  polluted  data.  We  now  take 
up  this  question.  Suppose  that  the  data  b  is  the  result  of  measurements  so  that 
instead  of  b  we  have  in  our  possession  a  corrupted  version  be  satisfying 
| | b  -  be | |  <.  e .  We  operate  on  the  vector  bE  to  obtain  the  approximations  xE 
given  by 

xg  =  U6(T)tV. 

Let  4>(e)  =  sup{  |  tU_(  t)  |  :  t  c  [0,||T||2]},  and  recall  that  <j>(b)  is  bounded  (by  (3)). 

P 

Lemma  3.  |  |Txg  -  Txg||  <.  e<j>(8). 

Proof.  T(x  -  xE)  =  TU-(T)T*(b  -  be),  therefore 

P  P  P 

llTxe  -  Tx^ll2  *  (T(xe  -  xE),xg  -  xE) 

•  (TUg(T)T*(b  -  be).xg  -  xE) 

=  (TUg(f)(b  -  be),T(xB  -  xE)) 
i  ♦(B)llb  -  bE||  ||T(xb  -  xE)|| 

<  ed>(e)  |  |Txb  -  Txe||.  # 

Suppose  now  that  g(B)  =  sup{ ] Ug( t) |  :  t  e  [0,||T||2]}.  We  note  that 

(6)  9(8)  "*■  “  $■+■<*> 

Indeed,  if  this  were  not  the  case,  then  there  would  be  a  constant  L  such  that 

|U  (t) |  <  L  for  all  t  and  B.  But  then  |tUg(t)|  <.  Lt  -*■  0  as  t  -*■  Q,  contradicting  (4). 

lemma  4.  |  |xg  -  xE|  |  <  e/g(8)4>(0)- 

Proof.  Since  xg  -  xE  ■  T*Ug(T)(b  -  bE),  we  have,  by  use  of  Lemma  3, 
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l|x6  -  xE|  |2  -  (x$  -  xc0,T\(T)(b  -  bE)) 

=  (T(xe  -  xE),UB(T)(b  -  bE)) 

1  e2< j>(6)g(6).  # 

Suppose  how  that  Pb  =  Tw  (we  could  also  use  the  other  hypotheses  considered 
above,  but  we  choose  to  consider  this  simple  case  to  illustrate  the  ideas).  By 
the  triangle  inequality  we  have 

l|x  -  X3I !  £  | Jx  -  xg | |  +  | |x6  -  xE||. 

Lemma  4  and  Proposition  4,  then  give 

Proposition  5.  If  Pb  =  Tw,  then 

|  | X  -  XE||  <  ( j  |w|  |a,(e.l)a,(0,O))J5  +  cfgUMe))**. 

The  first  term  on  the  right  hand  side  of  this  Inequality  goes  to  zero  as 
e  «.  However,  by  (6)  and  (4),  the  second  term  becomes  infinitely  large  as  B 
This  illustrates  the  classis  dilemma  in  the  numerical  treatment  of  ill-posed  pro¬ 
blems.  Even  If  computations  are  performed  exactly,  small  errors  In  the  data  may 
eventually  grow  and  overpower  the  approximations. 

EXAMPLES 

In  this  Section  we  will  consider  some  specific  choices  for  the  functions 
{U  (t) }  and  we  will  find  functions  u(B,v)  which  determine  rates  of  convergence. 

P 

The  index  set  S  in  all  examples  below  will  be  either  the  set  of  nonnegative  reals 
or  nonnegative  integers.  In  the  discrete  case,  the  parameter  B  will  be  denoted  by  n. 

As  a  first  example  we  consider  Showalter's  Integral  formula  [8]: 

T+b  =  /  exp(-uT)T  bdu. 
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The  functions  Ug  for  this  example  have  the  form 

8 

U  (t)  =  /  exp(-ut)du 
e  0 

and  may  be  motivated  in  terms  of  Borel  summability  [3].  It  is  not  difficult  to 
see  that  a  function  &>(b,v)  satisfying  (5)  is  given  by 

u>(8,v)  *  8  v  (\>  >  0). 

The  choice  Ug(t)  =  (t  +  8  ^)  ^  (8  >  0)  leads  to  Tychonov's  regularization  of 

order  zero  (see  [1]  and  [10]).  Here  one  can  readily  verify  that 

uj(8,v)  =  8  v  for  0  <  v  £  1. 

In  order  to  obtain  approximations  with  this  rate  for  v  >  1  we  may  use  extrapolated 
regularization  [5].  That  is,  for  a  given  8  >  0  we  set 

U*0)(t)  *  (t  +  B'1)"1 

and  define  Richardson  extrapolants  by 

U*j)(t)  =  (2jl4j_1)(t)  -  U*j_1)(t))/(2J"  -  1),  j  =  1,2 . 

It  Is  not  difficult  to  show  (see  [5,  Lemma  2.1])  that  for  k  *  0,1,2,... 

tk+1|l  -  tU^k)(t)  |  =  n  (-^ - ) 

B  i=0  216t  +  1 

<  M 

Therefore,  for  the  kth  extrapolant  we  may  apply  Theorem  4  with 

w(8,k)  ■  6"k’1,  k  *  1,2,...,  to  obtain  the  rate  e’1^  (see  [5,  Theorem  3.2]). 

We  now  consider  some  iterative  regularization  methods.  Below,  a  will  be  a 
parameter  satisfying  0  <  o  <  2||T||  . 
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If  the  functions  Un(t),  n  =  0,1,2,...  are  defined 

U  (t)  =  a  1(1-  “t)k 

n  k=0 

then  (3)  and  (4)  are  satisfied  and  one  can  show  that 

nV|l  -  tUn(t)  j  =  nV|1  -  at | n+1 

is  uniformly  bounded.  From  this  we  find  that  the  rate  of  convergence  of  the 
iterative  process 

Xq  =  aT*b,  xn+1  =  (I  -  aT)xn  +  aT  b 

is  determined  by  the  function  ai(n,v)  =  n"v. 

Newton's  method  for  approximating  t-1  leads  to  the  sequence  of  functions  de¬ 
fined  by 

u0(t)  =  a‘  Un+1(t)  =  un(t}(2  •  tun(t))* 

For  this  sequence  of  functions  it  is  not  difficult  to  see  that 

tv|l  -  tUn(t)|  =  0(2"vn)  for  v  >  0. 

Therefore  the  rate  of  convergence  of  the  corresponding  iterative  method  is  deter¬ 
mined  by  the  function  w(n,v)  =  2*vn. 

Showalter  and  Ben-Israel  [9]  have  extrapolated  on  the  previous  method  to  obtain 
methods  with  a  higher  rate  of  convergence.  For  a  positive  Integer  p  >  2  they  define 
the  hyperpower  methods  In  terms  of  the  sequence 

u0(t)  •  a.  ■  U„(t)  V  (1  -  tun(t))k. 

For  these  methods  the  results  above  may  be  used  to  obtain  the  convergence  rate 

0(p"vn). 
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In  [2]  the  following  Krayanev-type  method  [7]  is  studied 

x0  =  °’  ^xn  +  6"lxn  =  6"lxn-l  +  T*b  >  °)- 

One  can  verify,  as  in  the  first  iterative  example  above,  that  the  function 
w(n,v)  *  (enp  determines  a  rate  of  convergence. 

The  iterative  method 

xQ  =  T*b,  xn+1  =  xn  +  (T*b  -  Txn)/(n  +  2), 

was  investigated  in  [6].  Following  the  analysis  given  in  [6]  one  can  show  that 
the  rate  of  convergence  of  this  method  is  governed  by  the  function  w(n,v)  * 

(log  n)"v. 
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THE  PARAMETER  CHOICE  PROBLEM  IN  LINEAR  REGULARIZATION: 

A  MATHEMATICAL  INTRODUCTION 

C.  W.  Groetsch 

1 .  INTRODUCTION 

The  concept  of  well-posedness  for  an  equation  of  the  form 

Tx  -  b  (1.1) 

was  formulated  at  the  turn  of  the  century  by  Hadamard.  The 
equation  (or  problem)  (1.1)  is  said  to  be  well -posed  in  the  sense 
of  Hadamard  if  for  each  b  the  equation  has  a  unique  solution 
and  this  solution  depends  continuously  on  t>.  If  T  is  a 
transformation  from  a  topological  space  X  into  a  topological 
space  Y,  then  the  idea  of  well-posedness  may  be  dissected  by 
noting  that  the  following  three  conditions  on  the  triple  (T.X.Y) 
are  required: 

(I)  for  each  b  c  Y,  equation  (1.1)  has  a  solution, 

(II)  the  solution  x  Is  unique, 

and  (111)  the  mapping  b  •*  x  Is  continuous. 
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In  this  formulation  it  is  evident  that  the  weVl-posedness  of 
(1.1)  depends  not  only  on  T  but  is  also  intimately  connected 
with  the  spaces  X  and  Y  and  the  topologies  that  they  carry. 
Obviously,  if  the  space  Y  Is  too  broad,  or  if  X  is  too 
narrow,  then  (i)  cannot  be  satisfied.  While  if  X  is  too  large 
then  (ii)  will  not  hold.  Moreover,  if  the  topology  of  Y  is 
too  weak  or  that  of  X  too  strong,  then  ( i  1, i )  will  be  violated. 

In  this  exposition  we  shall  restrict  our  attention  to  the 
case  In  which  X  and  Y  are  Hilbert  spaces  and  T  is  a 
bounded  linear  transformation  from  X  to  Y.  With  this  con¬ 
text  In  mind,  let  us  assume  for  the  moment  that  for  a  given 
b  c  Y  equation  (1.1)  has  a  solution.  Condition  (ii)  will  then 
hold  If  and  only  If  N(T)  ■  (0)  (N(T):=  (xtX:  Tx*0}  is  the 

null space  of  T).  If,  however,  N(T)  f  {0},  then  (1.1)  has 
Infinitely  many  solutions,  namely  all  vectors  of  the  form  x  +  y 
where  x  is  a  particular  solution  and  y  e  N(T).  This  family 
of  solutions  Is  clearly  a  closed  convex  set  and  therefore  con¬ 
tains  a  unique  member  of  smallest  norm.  An  elementary  geometri¬ 
cal  argument  characterizes  this  minimal  norm  solution  as  the 
unique  solution  which  is  normal  (i.e.  orthogonal)  to  N(T).  We 
will  therefore  call  the  minimal  norm  solution  of  (1.1)  the 
normal  solution.  Since  the  normal  solution  is  unique  we  see 
that  (11)  Is  satisfied  if  we  replace  "solution"  by  "normal 
solution". 

The  normal  solution  exists  of  course  only  if  b  t  R(T) 
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(R(T):»  {Tx:xeX}  Is  the  range  of  T).  If  b  /  R(T)  one  might 
reasonably  broaden  one's  notion  of  solution  by  seeking  a  vector 


In  X  which  minimizes  the  functional 
f(u)  «  ||Tu-b||  . 

It  Is  a  simple  matter  to  show  that  the  minima  of  f  are 
precisely  the  solutions  of  the  equation 

Tx  -  Pb  (1 .2) 

where,  here  and  henceforth,  P  denotes  the  orthogonal  projec¬ 
tion  of  the  Hilbert  space  Y  onto  R(T),  the  closure  of  the 
range  of  T.  Any  solution  of  equation  (1.2)  will  be  called  a 
pseudo-solution  of  (1.1).  Equation  (1.2)  has  a  solution  if  and 
only  If  Pb  e  R(T)  which  is  immediately  seen  to  be  equivalent 
to  b  £  R(T)  +  R(T) 'L .  Assuming  that  (1.2)  is  solvable,  i.e. 
that  b  £  R(T)  +  R(T)1,  we  will  call  the  normal  solution  of 
(1.2)  the  pseudo-normal  solution  of  equation  (1.1).  The  opera¬ 
tor  which  associates  with  each  b  £  R(T)  +  R(T)1  the  pseudo¬ 
normal  solution  of  (1.1)  is  called  the  (Moore-Penrose) 
generalized  inverse  of  the  operator  T.  The  generalized  in- 

x  J- 

verse  of  T,  which  Is  denoted  by  T  ,  has  domain  /)(T  ):■ 

R(T)  +  R(T)1,  which  is  a  dense  subspace  of  Y,  and  is  a  closed 
linear  operator  (see  e.g.  [20]).  We  therefore  see  that  the 
triple  (T,N(T)X,^ (T+))  satisfies  (1)  and  (ii)  if  we  are 
trilling  to  extend  our  notion  of  "solution"  to  "pseudo-normal 
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solution".  Moreover,  the  "solution  operator"  in  this  context 
Is  the  generalized  Inverse  T+.  For  further  information  on 
generalized  Inverses  the  reader  may  consult  (6],  [36],  [50]  and 
[20]. 

The  scheme  of  things  developed  above  does  not  however 
satisfy  criterion  ( i i 1 )  (if  it  did  there  would  be  no  linear 
Ill-posed  problems  and  this  paper  would  not  exist!).  Indeed  it 
Is  not  difficult  to  show  that  the  operator  '(+  is  continuous 
if  and  only  if  R(T)  Is  closed  (see  e.g.  [20])  and  in  the  im¬ 
portant  example  of  Fredholm  integral  equations  of  the  first  kind 
the  range  Is  closed  if  and  only  if  the  kernel  is  degenerate,  i.e. 
only  quite  rarely.  We  are  therefore  led  to  consider  stable 
approximations  to  the  pseudo-normal  solution.  A  natural  way  to 
do  this  is  to  find  bounded  linear  operators  which  approximate 
T+  In  the  polntwlse  sense.  One  approach  to  this  problem  is  to 
view  pseudo-solutions  as  solutions  of  the  so-called  normal 
equation 

T*Tx  ■  T*b  ,  (1.3) 

where  T*  is  the  adjoint  of  T.  It  is  easy  to  show  that  equa¬ 
tions  (1.2)  and  (1.3)  have  the  same  solution  sets  (see  e.g. 

[20]).  If  we  denote,  as  we  shall  in  the  sequel,  the  self-ad¬ 
joint  operator  T*T  by  T,  then  we  see  that  the  set  of  all 
pseudo-solutions  Is  T"^  T*b.  One  might  therefore  reasonably 
attempt  to  form  stable  approximations  to  a  pseudo-solution  by 
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using  vectors  of  the  form  U  (T)T*b,  where  (U  (t)}  ..  is  a 

o  a  «>0 

family  of  continuous  real-valued  functions  on  the  spectrum  of  T 
(which  Is  contained  in  [0,||T||  ])  which  approximates  in  some 
sense  the  function  t  -*■  t_1 .  In  [20],  it  is  shown  that  if 
|tU  (t)|  is  uniformly  bounded  and  U  (t)  •*  t."1  as  a  -*•  0 

O  Cl 

for  each  t  >  0,  then 

x  :=  U  (T)T*b  -*■  T+b 
a  a 

as  a  -*•  0  for  each  b  e  ,£>(T+).  We  shall  call  such  a  family  of 
operators  {Ua(T)T*}  a  regularizor  of  equation  (1.1).  This 
method  of  constructing  regularizors  via  spectral  theory  was 
previous1,/  investigated  in  a  somewhat  different  context  by 
Bakushinskii  [5].  We  remark  that  the  condition  b  e  Z9(T+) 

Is  also  a  necessary  condition  for  the  convergence  of  the 
regularizing  algorithm.  In  fact,  if  b  /  £XT+)  then  it  can  be 
shown  that  {Ua(T)T*bl  has  no  weakly  convergent  subnet  [21]. 

From  the  practical  viewpoint  the  crux  of  the  difficulty 
with  Ill-posed  problems  Is  the  fact  that  the  right  hand  side  is 
typically  the  result  of  measurements  and  is  therefore  only 
approximately  determined.  Since  the  solution,  operator  is  in 
general  discontinuous.  It  then  happens  that  small  errors  in  the 
right  hand  side  can  lead  to  large  variations  in  the  computed 
solution.  Specifically,  If  6  is  some  approximation  to 
b  c  <£{T+)  and  the  approximations  x  are  defined  by 
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:■  Ua(i)T*b  , 

then  one  can  establish  the  bound 

I  IVT+bl  I  1  HVT+bN  +  Nb-6|  |  (Mg(a))1/2  (1.4) 

where  |tU  (t)|  <_ M  and  g(a)  =  sup  |U  (t)|  (see  [21]). 

t  a 

As  has  already  been  noted,  the  first  term  on  the  right  hand  side 
of  this  inequality  tends  to  zero  as  a  -*•  0,  however,  the 
hypotheses  on  the  functions  (Ua(t)}  imply  that  g(a)  ■+  ® 
as  a  -*•  0.  Therefore,  this  bound  illustrates  the  typical  di¬ 
lemma  In  the  numerical  analysis  of  ill-posed  problems:  the 
error  consists  of  two  components,  one  of  which  is  independent 
of  the  measured  data  and  tends  to  zero  while  the  other  in¬ 
creases  without  bound  for  a  fixed  level  of  error  in  the  input 
data.  Too  small  of  a  choice  of  a,  for  a  fixed  level  of 
error  In  the  data,  will  consequently  cause  the  quality  of  the 
computed  approximations  to  be  debased  under  tne  influence  of 
this  error.  The  effective  numerical  treatment  of  such  problems 
therefore  requires  criteria  which  relate  the  regularization 
parameter  to  the  error  level  In  the  input  data  in  such  a  way 
that  as  the  error  tends  to  zero  the  regularized  approximations 
x^  tend  to  the  pseudo-normal  solution.  The  choice  of  a  proper 
parameter  is  therefore  In  the  words  of  Baker  [3],  "a  practical 
detail  of  great  relevance". 

Thus  the  first  step  In  the  numerical  solution  of  (1.1) 
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Involves  two  essential  choices:  the  selection  of  a  regularizor 
and  the  choice  of  a  criterion  for  selecting  the  regularization 
parameter.  The  regularizor  which  has  been  studied  most  exten¬ 
sively  is  the  so-called  Tikhonov  [61]  regularizor  given  by 

Ufl(t)  =  (t+a)"1 

(see  also  [53]  and  [66]).  Our  aim  in  the  sequel  is  to  give  a 
theoretical  presentation  of  various  parameter  choice  strategies. 

II .  A  Priori  Parameter  Choices 

We  shall  suppose  that  b  e  «£)(T+)  and  b6  is  some 
measured  approximation  to  b  satisfying  |  [ b-b6 1 1  <_  6  (this 
will  be  relaxed  below).  The  actual  approximations  with  which  we 
are  forced  to  work  are  given  by 

x6  :«  U  (T)T*b5 
a  a 

while  the  "idealized"  approximations  are 

x  U  (T)T*b  . 
a  ci 

The  earliest  parameter  choice  criteria  involved  some  a  priori 
choice  of  a  depending  on  the  error  level,  say  o  ■  a(6), 
such  that 

l|x‘(s)-Ttb||  .0  as  s  -  0  . 

Of  course,  the  optimal  choice  would  be  some  value  a  *  oQ 
satisfying 
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I  |xf  -T+b||  =  inf  1 |x^-T+b| |  , 

aO  o>0  a 

but  such  a  value  of  the  parameter  is  for  practical  purposes 
impossible  to  find.  Alternatively,  we  might  reconsider  the 
bound  (1.4)  to  obtain 

||x«-  T+b||  <  ||xa-T+b||  +  6(Mg(a))1/2  (2.1) 

and  seek  a  quasi-optimal  value  of  a,  that  is  a  value  which 
minimizes  the  right  hand  side.  The  difficulty  here  is  that 
even  an  asymptotic  bound  for  the  quantity  |[xa-T+b||  usually 
requires  information  on  the  unknown  vector  b  which  is 
generally  unavailable  (see  [21],  [22]).  Kockler  [33]  has  shorn 
however  that  in  the  case  of  Tikhonov  regularization  for  matrices 
It  may  be  possible  to  find  an  a  posteriori  estimate  of  the  quasi- 
optimal  parameter. 

Leaving  aside  questions  of  optimality,  we  see  from  (2.1) 
that  the  method  converges  if  we  simply  choose  a  value  aU) 
of  a  such  that  a( <5 )  ■+•  0  and  6 (Mg (a ( 6 ) ) )  ^  -*-0  as  6  0. 

In  the  special  case  of  Tikhonov  regularization,  i.e.  Uft(t)  = 
(t+a)'1,  we  have  M  =  1  and  g(o)  =  a"1  and  therefore  a  suf¬ 
ficient  condition  for  convergence  is  a  -*•  0  and  6/ «/qT  -*•  0 
as  6  -»■  0.  Phrasing  this  slightly  differently,  if  we  choose 
positive  functions  Bj(6)  and  e2(fi)  which  converge  to  zero 
as  6  ■*  0  and  take  a  so  that  82U)  La  L 

that  Is, 


« 
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AfijU)  i  a  <  62(6)  ,  (2.2) 

then  convergence  is  assured.  The  formulation  (2.2)  appears 
in  Tikhonov  and  Arsenin  [64]. 

For  the  remainder  of  this  paper  we  will  be  concerned  ex¬ 
clusively  with  the  Tikhonov  regularizor.  During  the  past  decade 
and  a  half  the  analytical  theory  of  Tikhonov  regularization  has 
been  developed  to  a  very  high  level.  The  success  of  this 
effort  Is  due  largely  to  the  fact  that  the  Tikhonov  regularizor, 
unlike  the  general  class  of  regularizors  considered  in  the  in¬ 
troduction,  possesses  a  very  simple  and  natural  variational 
property.  Recall  that  our  goal  Is  to  approximate  the  pseudo- 
normal  solution,  that  Is,  the  vector  x  satisfying 

Tx  -  Pb 

for  which  | | x | |  is  minimal.  In  other  words,  we  wish  to 

2  2 
minimize  |  jx 1 1  subject  to  the  constraint  ||Tx-Pb||  =  0. 

Since  I-P  Is  the  orthogonal  projector  of  Y  onto  R(T)J' 

we  have 

||Tx-b||2  -  | | Tx-Pb | |2  +  | | ( I -P)b | | 2 

?  2 

and  hence  we  wish  to  minimize  ||x||  *  subject  to  | | Tx-b | | 

2 

■  | | ( I-P)b } |  .  Classical  Lagrange  multiplier  theory  leads  us 
to  consider  the  functional 

M|Tx-b||2  *  iwi2 
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or  equivalently,  the  functional 

£  [x;b,a]  =  |  ( Tx-b |  | ^  +  <x||x||^  (a>0)  .  (2.3) 

When  no  confusion  ensues  we  will  simply  refer  to  this  functional 
as  £  [x] .  Note  that  <JF  [x]  is  a  positive  quadratic  functional 
and  hence  has  a  unique  minimum  given  by 

grad  $  [x]  ■  0  . 

But 


grad  J  [x]  *  2T*(Tx-b)  +  2ox 

and  therefore  the  unique  minimum  xa  of  J  [x]  is  given  by 

xa  *  (T+al)’1  T*b  (2.4) 

(recall  that  T  :=  T*T) .  The  Tikhonov  regularized  approximation 
to  the  pseudo-normal  solution  of  (1.1)  is  therefore  precisely 
the  mlnimlzer  of  the  functional  (2.3).  This  opens  the  door  for 
the  application  of  powerful  variational  techniques  in  the  study 
of  these  approximations. 

We  now  present  a  minor  variant  of  a  classical  result  of 
Tikhonov  [62]  which  shows  that,  at  least  for  weak  convergence, 
condition  (2.2)  on  the  parameter  is  not  necessary.  We  shall 
suppose  that  b  e  29(T+)  and  b{  satisfies  j  | Pb-Pb6 1 1  <_  6. 
This  last  condition  allows  b  and  b5  to  differ  by  an  arbitra¬ 
rily  large  component  in  R(T)1  ■  N(T*),  a  reasonable  assumption 
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since  any  such  component  is  annihilated  by  T*  in  the  regular 
ization  process.  In  this  regard,  note  that  by  "lumping"  any 
component  of  b  which  lies  in  R(T) into  the  vector  b6  we 
may  as  well  assune  that  b  e  R(T),  that  is,  the  condition 
b  «  £)(T+)  and  !|Pb-Pb6||  <_  fi  may  always  ipe  replaced  by 
b  £  R(T)  and  |  |b-Pb5l|  <_ 6  . 

Before  proving  Tikhonov's  theorem,  we  take  note  of  the 
following  simple  fact. 

Lemma  2.1.  For  any  b  «  Y,  xa  *  (T+al)"1  T*b  «  N(T)X  . 

Proposition  2.2.  Suppose  b  «  ^(T+),  1 l Pb-Pb6 l 1  1  6  and 

that  for  some  positive  constants  and  C^,  a  satisfies 

2  2 

<  o  <  C2i  .  Then 

6  W  + 

x  -*■  T  b  (weak  convergence) 

O 

as  6  -►  0,  where  x{  *  (T+al)"1  T*b6  . 

a 

Proof.  Let  x  *  T+b.  Since  x5  minimizes  i![[*;b5,a]  and 

ICy;b‘.a]-  l|Ty-bsU2*a||y|l2 

-  I  |Ty-Pbsl  |2  *  ||U-P)US||2  +  a||y||2 

■  ICy;Pb6.«]  ♦  | |(l-P)b*| |2 

It  follows  that  x5  minimizes  5  [*;Pbfi,o].  Therefore 
ot 

max{||Tx^Pb6!|2,C162||x£||2}  ll  [x*;Pb6,a] 

<  $[x;Pb6,o]»  l lTx-Pb6I I2  +  a l Ml2  <_«2  +  C262||x||2 
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It  follows  that 

Tx5 -*■  Pb  as  6  0  and  { 1 1 x 6 1 1 ^ }  Is  bounded, 

a  1  1  a 1  1 

For  any  sequence  T 6n >  of  positive  numbers  converging  to  zero 

we  therefore  have  a  subsequence,  again  denoted  by  {6  }, 

6  w  2  2 

such  that  x  u  for  some  u  c  X,  where  C,6  <  a  <  C06„  . 

a  l  n  —  n  —  L  n 

n 

But  since  T  is  weakly  continuous,  this  implies  that 
6  w 

Tx  n  -*•  Tu  and  hence  Tu  *  Pb,  i.e.  u  is  ,j  pseudo-solution. 
an  5 

Since  x  n  e  N(T)  for  each  n  (by  Lemma  2.1)  and  since  N(T) 
“n 

is  closed  and  convex  (and  hence  weakly  closed),  we  have 

u  €  N(T)1.  Therefore  u  a  x,  the  pseudo-normal  solution,  and 

for  each  sequence  (6  }  converging  to  zero  there  is  a  subse- 
6  w  &  w 

quence  with  x  -*■  x.  From  this  it  follows  that  x  -*•  x  as 

a_  a 

n 

6  -  0.  # 


The  essential  features  of  the  argument  above,  i.e.  the  use 
of  the  variational  principle  to  bound  the  approximations  and 
then  the  exploitation  of  the  weak  continuity  of  the  operator, 
will  be  used  several  times  in  the  sequel. 

It  should  be  remarked  that  the  weak  convergence  of  the 
regularized  approximations  may  be  quite  satisfactory  if  X  has 
a  sufficiently  strong  norm.  For  example,  if  we  take  for  X 
the  Sobolev  space  W^[a,b],  being  the  completion  of  the  space 
C*[a,b]  with  respect  to  the  norm 
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then  weak  convergence  of  a  sequence  of  smooth  functions  implies 
the  uniform  convergence  of  the  sequence.  The  regularized  ap¬ 
proximations  xa  in  the  space  W^[a,b]  are  called  pth  order 
regularized  approximations  by  Tikhonov. 

A  number  of  early  papers  on  regularization  use  a  priori 
choices  of  the  regularization  parameter.  Tikhonov  and  Glasko 

[65]  have  given  a  number  of  illustrations  of  the  use  of  a  choice 
2 

of  the  form  a  =  C6  in  the  numerical  solution  of  integral 

equations  of  the  first  kind  by  second  order  regularizors . 

Franklin  [11]  suggests  the  choice  a  =  6  /u>  if  u  is  an  a 

priori  bound  on  the  norm  of  the  solution.  Ivanov  [28]  has  shown 

that  the  condition  6  *  o(/a)  is  necessary  and  sufficient  for 

the  strong  convergence  in  L  of  regularized  approximations 

for  Fredholm  Integral  equations  of  the  first  kind,  while  the 

condition  6  =  0(/x)  is  necessary  and  sufficient  for  weak  con- 
o 

vergence  in  L  .  Khudak  [30]  shows  that  if  the  kernel  of  the 

integral  operator  K  is  continuous  and  if  the  solution  lies  in 

2 

R(K*),  then  the  regularized  approximations  in  L  converge 
uniformly  if  6  =  0(a).  For  a  positive  definite  self-adjoint 
compact  operator  A,  Bakushinskii  [4]  has  studied  the  "simpli¬ 
fied"  regularization  procedure 

xj  -  (A+al)”1  b6. 

If  b  e  R(A)  and  j|b-b5||  <_5,  he  shows  that  a  sufficient 
condition  for  convergence  is  6  ■  o(a)  (see  also  Ivanov  [27] 
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and  Khudak  [31]).  A  more  general  regularization  procedure  of 
this  type  was  investigated  recently  by  Franklin  [12]  from  a 
variational  point  of  view.  Finally,  we  mention  that  Tikhonov 
[63]  and  others  have  given  criteria  for  an  a  priori  choice  of 
the  parameter  in  the  regularization  of  certain  nonlinear  pro¬ 
blems.  In  particular,  see  Pyazantseva  [54]  and  Al’ber  [1]  for 
recent  results  on  the  regularization  of  problems  involving 
nonlinear  monotone  operators. 


Ill .  Morozov's  Discrepancy  Principle 

The  discrepancy  principle  of  Morozov  is  based  on  the 
reasonable  view  that  the  quality  of  the  results  of  a  computa¬ 
tion  cannot  be  greater  than  the  quality  of  the  input  data.  To 
quote  from  Morozov  [45]  "the  magnitude  of  the  error  must  be  in 
agreement  with  the  accuracy  of  the  assignment  of  the  input  data 
of  the  problem".  The  discrepancy  principle  in  its  simplest  form 
states  that  if  b  e  R(T)  and  1 1 b-b6 1 1  <_  6  <_  |  |b5  j  | ,  then 
there  Is  a  unique  value  of  the  parameter  a,  which  we  shall  call 
a(s),  such  that 

l,Tx«(s)'bll  ■  8  • 

Here  is  the  Tikhonov  regularized  approximation  (2.4). 
Moreover,  as  6  0,  Xg(5)  x*  where  x  is  the  normal  solu¬ 

tion  of  (1.1).  This  result.  In  a  slightly  altered  form,  was 
also  published  Independently  by  Arcangeli  [2]  (see  also  Ivanov 
[26])  and  was  to  a  certain  extent  implicit  in  an  algorithmic 
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form  In  the  works  of  Phillips  [53]  and  Twomey  [66]. 

In  this  section,  we  will  prove  a  discrepancy  principle 
under  the  assumption  that  b  e  $(T+)  which  we  have  seen  is 
the  least  we  can  expect  of  the  data  in  order  to  obtain  conver¬ 
gence  even  in  the  error  free  case.  We  shall  suppose  that 
2 

| | T | |  <_  <  and  that  the  available  data  consist  of  a  vector 

b5  c  Y  satisfying  |  | Pb-Pb 6 1  J  <_  6  (note  again  that  this 
allows  b  and  b5  to  differ  by  an  arbitrarily  large  component 
In  R(T)1  and  therefore  we  could  just  as  well  assume  that 
b  c  R(T) ) .  Our  measure  of  the  "discrepancy"  in  the  approxima¬ 
tion  Is  specified  by  the  function 

p(a)  :=  ||Tx«-T*bS||2  . 

A  direct  application  of  Morozov's  principle  to  approximating  the 
pseudo-normal  solution  would  call  for  monitoring  the  size  of 
1 1 Tx*-Pb<5 1 1 .  However,  the  projection  P  is  not  available  in 
the  computations  which  necessitates  a  slight  modification  in  the 
method  such  as  the  one  we  now  present. 

Lemma  3.1 .  p  Is  an  Increasing  continuous  function  of  the 

6  2 

positive  variable  a.  Moreover,  p(»)  =  ||T*b  ||  and 
p(0)  »  0. 

Proof.  First  note  that  x*  *  0  and  hence  p(<»)  =  ||T*b<5||2. 

If  we  denote  the  spectral  resolution  of  the  positive  self-ad¬ 
joint  operator  T  by  {Ex> »  then  we  have  the  representation 
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l|ExT*b6||2  . 


from  which  it  follows  that  p  is  increasing  and  continuous. 

Also,  p(0)  =  J  IPn(T)  T*b5H2  =  0  since  T*b<5  £  N(T)\  # 
Note  that  If  y  is  any  pseudo-solution  then 
||Ty-T*b6||2  „  ||T*Pb-T*Pbfi||2  <  *5  2  .  (3.1) 


Moreover,  a  vector  z  of  minimal  norm  satisfying  the  above 
Inequality  also  satisfies 

j|fz-T*b6||2  -  k 6Z 

for  If 

1  /2 

a  :*  ||Tz-T*b4|l  <  k1/,25  and  y  =  min  jl , 

then  the  vector  y  *  (l-y)z  has  norm  smaller  than  ||z||  and 
satisfies  (3.1).  This  suggests  considering  the  regularized 
approximations 

x6  =  (T+al)'1^6 
a 

where  the  parameter  a  is  chosen  according  to  the  condition 

p(«)  *  l!Tx£-T*b6||2  -  <62  .  (3.2) 

From  the  lemma  it  follows  that  if  k62  <_  jjT*bfi||2  ,  then 

2 

there  is  a  unique  number  a(5)  satisfying  p(a (6))  ■  <6  . 
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The  following  result  shows  that  this  choice  of  the  parameter, 
which  relates  the  discrepancy  in  the  approximation  to  the  level 
of  error  In  the  input  data,  gives  a  stable  approximation  scheme 
for  computing  the  pseudo-normal  solution. 

Proposition  3.2.  Suppose  b  £  <£)(T+),  ||Pb-Pb6||  <_  &  and 
te<5^  £  |  |T*b6|  |2  where  ||T||2<k  ,  then  *  T+b  as 

6  0  if  p  ( cx(  6 ) )  =  tcfi2. 

Proof.  We  first  recall  that  the  Tikhonov  approximation  xa  » 
(T+oI)”H*y  minimizes  the  functional  defined  by 

I  [z;Py,a]  =  ||Tz-Py||2  +  a||z||2  . 

Therefore,  if  x  =  T+b,  then 

^  Cxa(6);Pb6*o(fi^i-^tx;Pb6*^6^  . 

and  hence 

i  iTx«t«rPb<sn2  *  °(s)  1  ^(4>ii2  i 1  iTx-p»6i  i2  ♦  «<4>  i  i*i  i2 

•  ||Pb-Pb5||2  +  »(«)|]x||2  <  s2  ♦  a(«)||x||2  . 

However, 

llT<(5rPt'Sll2i«''l|T||2||Tx4({)-Pb<||2 
.  «"'||fx4(8)-T*bS||2  ■  «■’,(>(«))  ■  «2  • 

Substituting  this  in  the  Inequality  above,  we  have 

I !<(,)!  I  <  11*11 


(3.3) 


t 
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for  all  6  >  0.  Therefore  the  set  {x 5 / , % : 6>0 >  is  bounded 
and  hence  weakly  pre-compact. 

Suppose  now  that  { 6n }  is  any  sequence  of  positive  numbers 
converging  to  zero.  Then  there  is  a  subsequence,  again  denoted 

6  W 

by  {«n>,  with  xa^6  j  -*•  y.  Also, 


ll*Xa(«  )‘fx II  iH fxa(5  )-T*b  "11  +  I |T*b°a-Tx| l 
*  K1/26n  +  HT*b6n-T*b|| 

»  x1/26n  +  ||T*P(bVb)j| 

1  /2 

<  2<  '  s  -*-0  as  n  -*•  «  . 

—  n 

6  W  .6 

Therefore  x  V.  v  ■*  y  and  Tx  v.  ,  Tx  as  n  -►  ■>  .  It  now 
'fin;  a'6n' 

follows  as  In  proof  of  Proposition  2.2  that  y  >•  x  and 


x  /x»  •*  x  as  6  -*■  0. 

a(6) 


(3.4) 


o 

For  any  convergent  sequence  {||xo(5  j||)  we  have 

I  lx 1 12  -  11m  |<x,x  }>|  <  | |x| |  lim  | |x  J  ,|| 

n-M  '  n '  '  n  ' 


and  therefore 


|x|l  <  11m  Inf  ||xj(4)|| 
6  0 


But  by  (3.3) 


11m  sup  ||x«(a)||  I  [|x|| 

and  hence  ||x||  »  11m 
{•♦0 

proves  that 


•  This  combined  with  (3.4) 


i 
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1  im  x 
6-0  c 


U) 


=  x. 


In  using  the  discrepancy  method  it  would  be  helpful  to  have 
an  a  priori  bound  on  the  regularization  parameter.  Vinokorov 
[67]  has  provided  such  a  bound.  A  slight  modification  of 
Vinokorov's  argument  yields  the  following  result. 


Proposition  3.3.  If  the  parameter  a  Is  chosen  according  to 
criterion  (3.2)  above,  then 

a  <_  k3/2  6/(|  |T*b6J  |2  -  J/26)  . 


We  have  considered  the  discrepancy  method  only  in  a  very 
simple  context  but  we  wish  to  point  out  that  the  method  has  been 
highly  developed  by  Morozov  and  his  colleagues  (see  the  works  of 
Morozov  and  Goncharskii  et  al.  In  the  references). 

IV.  OTHER  A  POSTERIORI  METHODS 

The  discrepancy  method  is  but  one  of  a  number  of  a 
posteriori  strategies  which  have  been  proposed  for  choosing  the 
regularization  parameter.  In  order  to  motivate  some  of  these 
methods  we  recall  some  facts  about  the  accuracy  in  Tikhonov 
regularization.  It  Is  well  known  that  if  Pb  e  R(TT*)  then  the 
(error-free)  Tikhonov  method  converges  with  a  rate  a  '  (see 
e.g.  [29]).  However,  If  the  data  satisfy  stronger  assumptions 
methods  can  be  devised  with  a  corresponding  higher  rate  of 
convergence  (see  [21]  for  a  general  result  along  these  lines). 
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In  particular  what  we  might  recall  the  iterated  Tikhonov  method 
given  by 

Z  :»  o(T+aI)"^x  +  x 
a  a  a 

where 

x  :=  (T+aD'Vb 

a 

has  been  investigated  by  Dorofeev  [9]  and  King  and  Chill ingworth 
[32].  If  Pb  e  R(TT*T)  then  it  can  be  shown  that  za  con¬ 
verges  with  a  rate  a3>/2  [32].  Note  that 

^  -  -(T+oI)"2  T*b  -  -(f+oI)_1x 

Qa  a 

and  hence 

°  *  "  »(T+ol)-1xo  *  *0  -  za  . 

Therefore  a  choice  pf  a  which  minimizes 


tends  to  move  the  "cheap"  approximation  xa  closer  to  the 
generally  more  accurate  approximation  Zfl  .  In  the  presence 
of  erroneous  data  a  value  of  a  which  minimizes  (4.1)  is  called 
In  the  Soviet  literature  a  quasi-optimal  choice  of  the  parameter 
([64], [65]),  but  we  shall  call  it  a  choice  by  the  derivative 
criterion.  For  a  numerical  example  of  the  use  of  this  parameter 
choice  criterion  In  an  Inverse  problem  In  heat  conduction,  see 
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Glasko  et  al .  [13]. 

Another  approach,  Tikhonov  and  Arsenin's  ratio  criterion. 
In  our  context  advocates  maximizing  the  ratio 

||Tz*-T*b6||/||Tx*-T*b5||  , 

which  tends  to  make  the  residual  of  x6  as  small  as  possible 

a 

relative  to  the  residual  of  z5  . 

a 

We  now  present  some  arguments  of  Leonov  [38]  who  has  es¬ 
tablished  the  convergence  of  the  simplified  Tikhonov  method  in 
finite  dimensional  space  using  the  derivative  criterion  for  the 
choice  of  the  parameter.  We  shall  therefore  assume  that 
T  :  -*■  Rn  Is  linear.  Then  the  pseudo-solutions  of  equation 

Tx  *  b 

are  the  solutions  of 

Ax  -  y  (4.2) 

where  A  *  T*T  Is  an  n  by  n  symmetric  matrix  and 
y  *  T*b  £  N(A)1  *  R(A),  We  assume  a  vector  y6  e  Rn  is  on 
hand  satisfying  |(y-yfi||  <.  6  (Euclidean  norm)  and  we  seek  to 
approximate  the  normal  solution  of  (4.2)  by  the  simplified 
Tikhonov  approximation 

x6  :■  (A+aI)_1y5  *:  U  (A)y{  . 

a  a 

The  Ideal  simplified  Tikhonov  approximation  will  be  denoted  by 
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V  v1z* 

xa  :=  (A+oI)_1y  =  Ua(A)y  . 

Suppose  that  0  <  X-j  <_  *2  <_  . . .  <_  is  the  complete  system  of 

positive  eigenvalues  of  A  and  { u ^ ^  is  a  corresponding 
system  of  orthonormal  eigenvectors.  Since  y  e  R(A)  =  N(A)1 
we  have  a  representation  of  the  form 
r 

y  =  s  a.u.  . 
k=l  K  K 

Also,  for  suitable  coefficients  {a^}  we  have 

y6  =  w6  +  I  afu. 

k=l  K  * 

t 

where  w  c  N(A).  We  therefore  have 

«2i  IlyVll2  -  llw'll2  *  t  (ak-a*)2  >  ||ws||2  .  (4.3) 

k=l  K  K 

We  now  set 

dxs  2 

•  MaUa(A)2ys||2 

■  ||w4||2/o2  *  a2  £  (a?)2/(a«k)4  . 

k=l  K  K 

A  little  calculus  shows  that  F  has  a  smallest  relative 
minimum  which  we  denote  by  o'  (the  choice  of  a  by  the 
derivative  criterion).  From  the  fact  that  F*(o')  *  0  one 


9 
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Mw6||2/a'4  =  2  Z  [l  -  -Q-}  (aj)2/(a'+xj4  (4.4) 


=  2 

T. 

1  2 

k=l 

r 

I 

k=l 

_  ^  VO  L,  /  / 

k=l  K  K 

and  therefore  | I w5 | | ^/a 1 4  is  bounded  for  small  6.  Moreover 
the  equality  above  shows  that  a*  is  bounded  as  6-^0,  for 
otherwise  the  left  hand  side  would  become  negative  for  small  6. 
If  we  suppose  that  »'  <«,  then 

kf,  [’  -  '<•  V  i  k‘,  j'  - 

(a[)2/(mnk)4  . 

Since  a£  -*■  ak  as  6  -*•  0,  It  follows  from  (4.4)  that  for  6 
sufficiently  small 

«2  >  li"5!!2  i.  Ca'  4 

for  some  positive  constant  C  and  hence  a‘  -*•  0  as  6  -►  0. 

To  establish  the  convergence  of  x to  the  normal  solu¬ 
tion  x  we  note  that 


lxf'“xl  I  <  I  }x6 , -X  .||+|  |x  , -X I 
1  cl  —  a  a  a 


But  It  is  well  known  that  in  the  absense  of  error  in  the  right 
hand  side  the  simplified  Tikhonov  method  converges  to  the  normal 
solution.  Since  o'  -*■  0  as  6  -*■  0  we  therefore  have 
llx0*"x||  ■*  0  as  6  ■*  0.  Therefore,  we  need  only  estimate  the 
quantity  llxa'”xa'll  •  To  this  end,  we  give  the  following 


i 
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lemma. 

Lemma  4.1 .  Ilx^-xjl  <_  ||w6||/a  +  6/X^  . 

Proof.  Note  that 

' 

xo_xo  *  ua(A)(y6"y)  =  UJA)  +  (ak-akK 

k=l 

-  U  (A)  (vAaz5) 

a 

where 

p 

z‘  "  M  (ak‘1k,UK/Xl‘  ‘ 

But  by  (4.3), 

iiz6m2  =  ^  <aK>2/*k  <  m2  j,  (akS)2  <<>iV  • 

Since  |  |U  (A)  1 1  a"1  and  1 1 AU^fA )  1 1  <1  we  then  have 

ll**-\l|2  '  <Xa'VVA)"i>  *  <‘VX.'AU«<*>*‘> 

1  I|xj-XBI|  [||w6||/a  ♦  6/ ]  .  # 

Finally,  for  the  choice  a  *  o'  we  have  seen  that  there 

6  1/2 

Is  a  constant,  say  C^ ,  such  that  C-j  |  |w  1 1  '  <  a'  for  6 

sufficiently  small.  Substituting  this  in  the  lemma  and  using 
(4.3)  we  have 

I  |x£.-xa,  1 1  i  -0  as  «  -*•  0, 

which  proves: 

Proposition  4.2.  In  the  case  of  simplified  Tikhonov  regulariza- 
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tlon  in  a  finite  dimensional  space  the  choice  of  parameter  by 
the  derivative  criterion  gives  a  method  which  converges  to  the 
normal  solution. 

The  choice  of  the  regularization  parameter  by  the  deriva¬ 
tive  criterion  is  related  to  a  method  for  selecting  the 
regularization  parameter  investigated  by  Hilgers,  [24].  He  notes 
that  the  optimal  value  of  the  regularization  parameter  is  a 
value  which  minimizes  the  function 

e(a)  -  !|x^T+b[[2  . 

Differentiating  this  with  respect  to  o  we  havti 

,  .  dx6 

C'<„)/2  =  <xf-T+b,  -3J  >  . 


Therefore,  the  optimal  value  of  the  parameter, 
satisfy 


0  . 


an» 


should 


(4.5) 


If  we  assume  that  we  have  a  genuinely  ill-posed  problem  then 


Oq  >  0  and  this  condition  is  equivalent  to 

.6  I 


,  *  dx 

<xf  -T+b,o 
»0  da 


>  -  0  . 


a“af 


dx 


-j*.  <  ||x«-Ttb||  ||. 


9 


Since 
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we  may  view  the  derivative  criterion  as  an  attempt  to  approxi 
mately  satisfy  (4.5).  Hilgers  takes  a  different  tack.  Since 
TT+b  =  Pb  and 

dx5  ~  o  x  -ok 

=  "Ua(T)  i*b5  =  -T*Ua(T)2b6  . 

where  T  :■  TT*t  he  writes  (4.5)  as 


<Pb-Tx6  ,  U  (T)V>  =  0. 
a0  a0 


(4.6) 


All  of  the  quantities  on  the  left  are  computable  except  for  the 
projection  Pb.  But  if  b  c  R(T)  (or  if  T  is  self-adjoint 
and  one-to-one)  then  Pb  *  b  and  (4.6)  becomes 


<b-Txf  ,  U  (T)V>  =  0  . 

°0  a0 

But  of  course  the  vector  b  is  also  unavailable  and  therefore 
Hilgers  advocates  a  choice  of  a  which  minimizes  the  expression 


<b6-Tx6,U  (T)2b6>  . 
a  a 


Note  that  this  quantity  is  nonnegative  because 

b6-Tx6  -  (I-TU  (f )T*)b6 
a  a 

-  (I-TUa(T))b6 

■  aU  (T)b6 
a 


and  therefore 


THE  PARAMETER  CHOICE  PROBLEM  IN  LINEAR  REGULARIZATION  27 
We  might  also  notice  that  this  equality  also  gives  the  bound 

<b5-Tx£.Ua<T)2b6>  <  (||b6||/a)2 

*  -1 

since  | | U  (T) | |  <_ a"  .  The  use  of  this  method  of  choosing  the 
parameter  has  been  tested  on  sample  problems  in  [23]  and  [24]. 

It  should  be  remarked  that  the  choice  of  the  parameter  by  Hi 1  - 
ger's  criterion  requires  an  additional  inversion  as  (T )  is 
not  obtainable  directly  from  Ua(T). 

One  can  also  verify  the  convergence  of  simplified  Tikhonov 
regularization  in  a  finite  dimensional  space  with  Hilger's 
choice  of  the  parameter.  The  essential  point  is  to  find  an 
upper  bound  for  ||w6||  in  terms  of  the  parameter  and  use 
Lemma  (4.1).  To  this  end,  we  will  use  the  notation  established 
above  In  the  discussion  of  the  simplified  regularization  method 
for  the  equation  Ax  3  y. 

We  will  denote  by  on  a  value  of  a  which  minimizes  the 
function 

F(o)  *  a<U  (A)y6,U  (A)V> 

a  a 

-  |  |w6|  |2/a  +  a  [  (a£)2/(anj3  . 

k=l  K  * 

Using  F'  (<» j^)  ■  0,  we  find  that 
2|l«S||2/«h  (»k)2/»k  • 

Therefore  for  4  sufficiently  snail,  we  have  a.  >_  C^l | w4 1 
for  some  positive  constant  C2-  As  in  the  proof  of  Proposition 
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4.2  we  may  now  conclude  that 

x6  +  x  as  6  -*  0  . 

°h 

V.  IVANOVS  CONSTRAINED  PSEUDO-SOLUTIONS 

Ivanov  and  his  colleagues  ([25], [8])  have  studied  genera¬ 
lized  solutions  of  the  linear  operator  equation 

Tx  -  b  (5.1) 

which  are  required  to  satisfy  a  certain  norm  constraint  (see  also 
Miller's  [41]  "method  4").  In  studying  the  parameter  choice 
problem  for  a  regularized  approximation  to  such  a  generalized 
solution  we  shall  employ  terminology  and  notation  which  is  more 
consistent  with  that  which  we  have  used  in  the  previous  sections 
rather  than  adhering  to  that  used  by  Ivanov. 

We  suppose  that  T  is  a  bounded  linear  operator  from  a 
Hilbert  space  X  into  a  Hilbert  space  Y  and  that  R  is  a 
fixed  positive  number.  The  closed  ball  in  X  of  radius  R 
will  be  denoted  by  BR,  l.e.  BR  ■  {xeX:  ||x||  R}  .  By  an 
R-pseudo-solution  of  (5.1)  we  mean  a  vector  x  c  8R  satisfying 

| | Tx-b | |  -  Inf  { j |Tz-b| |  :  z  e  BR)  . 

We  note  that  an  R-pseudo-solution  always  exists  since  BR  is 
weakly  compact  and  the  functional  f(z)  »  ||Tz-b||  is  weakly 
lower  semlcontlnuous.  It  is  useful  to  characterize  the  set  of 
R-pseudo- solutions  In  a  more  geometrical  way.  First  note  that 
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the  set  TBR  Is  weakly  closed  and  convex  and  therefore  strongly 
closed.  If  we  denote  by  Q  the  metric  projection  of  Y  onto 
TBR  ,  that  Is  if  Qy  is  the  unique  element  of  TBR  which  is 
nearest  to  '  y  t  Y,  then  Q  is  continuous.  It  follows  directly 
from  the  definition  that  x  Is  an  R-pseudo-solution  of  (5.1) 

If  and  only  If 

Tx  -  Qb. 

If  we  denote  the  set  of  all  R-pseudo-solutions  by  XR,  l.e. 

XR  *  {x  €  Br:  Tx  *  Qb}  , 

then  XR  Is  closed,  convex  and  nonempty.  The  set  XR  therefore 
has  a  unique  element  of  smallest  norm  which  we  will  denote  by 
xR  and  call  the  R-pseudo- normal  solution  of  (5.1).  The  term 
normal  Is  used  since 

XR  n  N(T)X  •  {xR}  . 

Indeed,  If  u  e  XR  and  u  *  ui  +  u2  £  N(T)  ®  NCT)1,  then  since 
2  2  2 

I  lull  *  I  |u^  1 1  +  |  lu2 1 1  we  see  that  u2  £  br  and  moreover 

2 

Tug  *  Tu  *  Qb.  Furthermore  ||u||  is  minimal  if  and  only  if 
■  0,  l.e.  u  €  N(T)i  . 

Note  that  If  equation  (5.1)  has  a  pseudo-solution,  that  Is 
If  b  «  ^)(T+),  then  xR  *  T+b  If  and  only  if  ||T+b||  <_  R  . 

Lemma  5.1.  If  ||T+b||  >  R,  then  ||xR||  *  R  . 

Proof.  First  note  that  b  i  Qb.  For  If  Qb  ■  b  then  b  ■  Tx 


1 
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for  some  x  e  BR  and  hence 


b  -  Pb  -  Tx 


Therefore  x  Is  a  pseudo-solution  and  hence 


R  <  l|T+b|!  <  |(x||  <  R  . 


Suppose  that  ||b-Qb||  ■  y  >0  and 


sy  :■  {ye  y  :  |Iy-b||  <  y>  . 


SY  n  TBR  =  {Qb}  , 


each  of  S  and  TBD  Is  convex,  and  S  has  an  interior. 
Therefore  by  the  Hahn-Banach  separation  theorem  there  is  a 
linear  functional  $  such  that  $(y)  1  for  y  e  TBR  and 

$(y)  >1  for  y  c  S  .  If  ||xD||  <  R  then  there  is  a  6  >  1 
such  that  fixR  c  Br.  We  then  have 

1  <  B  *  MQb)  *  B$(TxR)  *  ♦^gxR)  l1*  # 

Lenma  5.2,  The  functional  g(z)  *  |tT+z||  is  lower  semi- 
continuous  on  ZS>(T+). 

Proof.  Suppose  {yn>  c  £>(T+),  yn  y  and  ||T+yn||  <_m. 

Then  there  is  a  subsequence  (y_  >  and  a  z  e  X  with 

nk 

t  w 

TTy„  z  . 


t 
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Since  B  Is  weakly  closed  we  have  ( ( z [ (  <  m.  Since  T+  Is 
m  - 

a  closed  linear  operator.  It  Is  also  weakly  closed.  But 
w  +  w  + 

yn  -►  y  and  T  yn  -*■  z  then  implies  that  y  «  £>(T  )  and 
k  + 

T  y  *  z.  Therefore  ||T  y||  im.  # 

Lemma  5.3.  Suppose  b  e  «CS(T+)  and  ||T+b||  >  R.  If 
1 1 Pb-Pb6 1 1  then  for  all  sufficiently  small  6  there  is  a 

unique  a  =  a(6)  such  that  i|x^6j||  *  R  where 
x6  =  (T+cxirVb5  . 

Proof.  It  is  not  difficult  to  see  that  the  function 
g(a)  :=  I  |x*|  |  is  continuous,  decreasing  and  satisfies 
g(«)  *  0.  If  b5  l  £)(T+)  then  g(0)  =  »  (see  the  introduc¬ 
tion).  While  If  b6  €  /$)( T+)  then  g(0)  -  |  JT^6 1 1  = 

||T+Pb6||  >  R  for  6  sufficiently  small  by  Lemma  5.2.  The 
result  now  follows.  # 

Proposition  5.4.  If  b  c  £KT+),  ||T+b||  >  R  and  ||Pb-Pb6||<6  , 
then 


Proof.  Since  x{  minimizes  the  functional 

OL 

I[x;b6,a]=  ||Tx-b6|[2  +  al|x||2 
we  have  $  [x*;b6,a]  <  $  [xR;b6,a],  that  is, 

||Tx«-bfitl2  +  a|lx«U21||TxR-bi[|2  +  aR2  . 
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A  2  2 

For  the  choice  a  =  a(fi)  we  have  llx°(5)ll  “  R  and  hence 

iK(4)-bS|i2-l|TvbS||2‘  n«b-b4n2 

+  |  |Qb-b[  8S  i  +  0  > 


Therefore 


lim  sup  llTx^arb6[|  i  [!Qb-bl|  . 

6  -  0  a{6' 

Since  ^  l  lXa( 6)11^  b°unded*  ^or  any  sequence  { 6 n >  of  posi¬ 
tive  numbers  converging  to  zero  there  is  a  subsequence,  again 

6  W 

denoted  by  {$„ }  such  that  x  V.  »  -*•  x.  We  ma/  conclude  as  in 

n  o(V  6  w 

previous  proofs  that  x  c  N(T)1  n  BR  and  Tx^  ^  Tx  . 

Now  If  z  e  Y  and  ||z||  =  1,  we  have 

6n 

| <Tx-b, z> l  a  lim  |<Tx  7  \-b,z>(  <_t[Qb-b((  . 

n  ;  n' 

Therefore  ||Tx-b||  <_  jlQb-b||  and  hence  Tx  *  Qb,  i.e. 


x  €  XR  n  N(T)1  -  {xR>  . 

s  w  6 

We  now  have  x°(6)  xr  as  fi  0  and  iKui11 ' R 

■  | |xR | |  and  hence 

Xa(8)-XR  «  S’°-  1 

The  result  above  gives  a  parameter  choice  criterion  for 
approximating  the  R- pseudo -normal  solution  assuming  some  a 
priori  Information  on  the  pseudo-normal  solution,  namely  that  its 
norm  Is  greater  than  R.  If  the  pseudo-normal  solution  has  norm 
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less  than  or  equal  to  R  then  the  R-pseudo-normal  solution 
coincides  with  the  pseudo-normal  solution  and  any  of  the  para¬ 
meter  choice  criteria  from  the  previous  sections  may  be  used  in 
computing  the  regularized  approximations. 

VII.  CONCLUDING  REMARKS 

The  mathematical  theory  of  regularization  for  linear  ill- 
posed  problems  has  developed  very  rapidly  and  in  many  directions 
during  the  past  seventeen  years.  Our  presentation  of  the  para¬ 
meter  choice  problem  was  rather  narrowly  aimed  and  in  this  sec¬ 
tion  we  wish  to  take  our  last  opportunity  to  briefly  mention 
some  related  topics. 

The  parameter  choice  criteria  presented  above  are  theoreti¬ 
cal  In  that  they  do  not  indicate  how  the  parameter  is  chosen  in 
actual  computational  practice.  One  rough  and  ready  way  to  choose 

the  parameter  in  practice  is  to  perform  a  regularization  for 

2 

several  values  of  the  parameter,  say  a,  a/10,  a/10  ,  ... 
where  a  is  fixed  (and  not  "too  small").  One  then  takes  the 
value  of  the  parameter  which  comes  closest  to  satisfying  the 
chosen  criterion.  This  has  been  called  the  "selection"  method. 
Gordonova  and  Morozov  [19]  (see  also  [18])  have  investigated  the 
use  of  Newton's  method  as  a  more  sophisticated  procedure  for 
solving  numerically  for  the  chosen  regularization  parameter. 

The  point  has  been  made  above  that  the  great  analytical 
success  of  Tikhonov's  method  is  due  in  large  measure  to  its 
simple  variational  Interpretation.  There  are  other  types  of 
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regularization  methods,  for  example  Iterative  methods  ([34], 
[35],  [39])  In  which  the  role  of  the  regularization  parameter  is 
played  by  a  discrete  parameter  (e.g.  the  iteration  number)  and 
for  which  there  is  no  natural  variational  interpretation.  There 
seems  to  be  no  satisfactory  parameter  choice  criterion  for  such 
methods.  Ill-posed  problems  may  also  be  regularized  by  the  use 
of  expansion  methods  ([3],  [37],  [55])  or  finite  element  type 
methods  ([51],  [68],  [40])  which  also  depend  on  a  discrete 
parameter  (the  dimension  of  a  subspace). 

We  have  taken  a  purely  deterministic  point  of  view  in  this 
presentation  but  there  is  a  substantial  literature  on  regulari¬ 
zation  in  which  statistical  properties  of  an  error  distribution 
are  assumed  and  corresponding  stochastic  convergence  properties 
of  the  regularized  approximations  are  derived.  We  refer  the 
reader  to  [7],  [10],  [46]  and  [59]  for  details  and  references. 

Finally,  we  mention  that  Strahov  and  his  colleagues  [56], 
[57],  [58]  have  studied  a  somewhat  more  general  linear  ill- 
posed  problem  than  the  one  we  considered  above.  Namely,  they 
study  the  problem  of  constructing  the  values  of  a  closed  un¬ 
bounded  operator.  Since  the  generalized  Inverse  Is  In  general 
closed  and  unbounded  their  results  apply  In  our  context. 
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